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Abstract 

Background: Valid tools measuring characteristics of the school environment associated with the physical activity 
and dietary behaviours of children are needed to accurately evaluate the impact of initiatives to improve school 
environments. The aim of this study was to assess the validity of Principal self-report of primary school healthy 
eating and physical activity environments. 

Methods: Primary school Principals (n =42) in New South Wales, Australia were invited to complete a telephone 
survey of the school environment; the School Environment Assessment Tool - SEAT. Equivalent observational data 
were collected by pre-service teachers located within the school. The SEAT, involved 65 items that assessed food 
availability via canteens, vending machines and fundraisers and the presence of physical activity facilities, 
equipment and organised physical activities. Kappa statistics were used to assess agreement between the two 
measures. 

Results: Almost 70% of the survey demonstrated moderate to almost perfect agreement. Substantial agreement 
was found for 10 of 13 items assessing foods sold for fundraising, 3 of 6 items assessing physical activity facilities of 
the school, and both items assessing organised physical activities that occurred at recess and lunch and school 
sport. Limited agreement was found for items assessing foods sold through canteens and access to small screen 
recreation. 

Conclusions: The SEAT provides researchers and policy makers with a valid tool for assessing aspects of the school 
food and physical activity environment. 



Introduction 

Schools have been recognised as an important setting 
for promoting child healthy eating and physical activity 
[1]. Ecological frameworks and empirical research sug- 
gest that a number of school characteristics such as their 
economic, policy and socio-cultural environment can 
impact upon these behaviours [2], Moreover, the phys- 
ical environment of schools can significantly impact 
upon the nutrition and physical activity behaviours of 
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low-nutrient dense foods such as soft-drinks and confec- 
tionary through school canteens, vending machines or 
fundraising activities have been associated with children 
consuming a higher intake of kilojoules and saturated fat 
and a lower intake of vegetables and fruit [1,4,5,7,8]. 
Similarly, the availability of open spaces (e.g. courts or 
fields) [9]; playground markings [10]; fixed outdoor 
equipment such as climbing structures [10,11], the avail- 
ability of sports equipment [12] and provision of orga- 
nized games or sports during recess and lunch [13] have 
been associated with higher levels of child physical activ- 
ity at school. 

Given the suggested influence of the school environ- 
ment on children's diet and physical activity, encour- 
aging schools to create environments supportive of such 
behaviours has been a focus of Government obesity 
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prevention policies and programmes in a number of 
countries [14]. For example, some jurisdictions in The 
United States and Canada have prohibited the sale of 
foods which are high in sugar and saturated fats through 
vending machines and school meals and have mandated 
the teaching of physical education and the amount of 
class time dedicated to moderate-vigorous activities [1]. 
Similarly, a number of Australian States and Territories 
have released mandatory Government policies banning 
the regular sale of unhealthy foods and beverages from 
canteens, school events and fundraisers [15]. Further- 
more, they have stipulated the minimum number of 
hours children must participate in physical education 
and sport [16]. Specifically within New South Wales all 
Government schools are restricted from selling some 
sugar sweetened drinks and foods that are high in satu- 
rated fats and or sugar and or salt such as confectionary, 
fried foods and sweet and savoury snack foods [17]. In 
addition all Government primary schools are required to 
allocate 120 minutes of planned physical activity each 
week, inclusive of school sport, in Years 3 to 6 [16]. 

Valid tools to measure school environment character- 
istics are needed to accurately assess the nature and 
prevalence of nutrition and physical activity promoting 
characteristics of schools, the trends and changes over 
time in the prevalence of such characteristics, and to 
evaluate the impact of initiatives to promote school im- 
plementation of environmental changes. While direct 
observation represents a gold standard data collection 
method, for assessment of school environmental charac- 
teristics such procedures are expensive and impractical 
at a population level [18,19]. 

Population based assessments of school environment 
characteristics in Australia [20], the United States of 
America [21] and Canada [22] have mostly relied on 
Principal or School Administrators completion of tele- 
phone or paper surveys. Despite the use of such mea- 
sures, few have been validated [18]. For example, 
the validity of the nutritional and physical activity envir- 
onmental characteristics of schools in Australia, as 
reported in large, recurrent surveys of Australian school 
Principals [20] has not been reported. Similarly, the 
United States National School Health Policies and Pro- 
grams Study (SHPPS) of school health policies and prac- 
tices has reported the reliability but not the validity of 
the questionnaires measuring school practices [23]. An 
unpublished validation study of the Canadian School 
Health Environment Survey (SHES) [24] is one of the 
few studies that have reported the validity of school sur- 
vey tools. The report found high percent agreement 
(range 70%-100%) between direct observation and com- 
bined school administrator/ teacher "best response" for 
questions related to existing physical activity facilities 
and healthy eating programs and promotions. Lower 



agreement was found for availability of healthy vending 
machines and promotion of active transport (50.0% and 
66.7% percent agreement respectively). The 2007 study 
was limited by it being conducted in 14 schools, of 
which eight catered for primary school aged children, 
and each school being observed for one day [18]. More 
recently, two school physical activity environment audit 
tools developed in the United Kingdom have been 
validated by direct observation. In both cases the tools 
focussed only on the physical activity or sport environ- 
ment of schools [19,25] 

Given the limited evidence regarding the validity of 
self reported school characteristics regarding the promo- 
tion of student healthy eating and physical activity, we 
undertook a study to determine the validity of Principal 
reported nutrition and physical activity characteristics of 
primary schools. 

Methods 

Approval to conduct the study was obtained from 
Hunter New England Area Health Service (HNEAHS) 
Human Research Ethics Committee (no. 06/07/26/4.04) 
and The University of Newcastle (H-2008-0343). 

Design and setting 

A descriptive study, comparing cross sectional data from 
two measures of the healthy eating and physical activity 
environment of primary (children 5-12 years of age) and 
central schools (children aged 5-18 years of age) was 
undertaken. The study was conducted in the context of 
a large scale child obesity prevention initiative in the 
Hunter New England region of New South Wales 
(NSW), Australia [26]. 

Participants 
Principal self-report 

All primary schools in the study region were approached 
to participate in a 20-minute telephone survey of school 
environment characteristics related to the promotion of 
child healthy eating and physical activity. 

Observational data 

A sub-sample of schools participating in the telephone 
survey were approached to participate in the collection 
of observational data regarding school environment 
characteristics. 

Data collection procedures 
Principal self-report 

A database of all primary and central schools across the 
state was generated from information provided by rele- 
vant school bodies [27-29]. All schools were eligible to 
participate in the study other than special purpose 
schools catering for students with special needs, juvenile 
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justice or schools serving children who are hospitalized. 
Principals were sent a letter inviting them to participate 
in the study. Two weeks after receipt of the letter, Princi- 
pals were telephoned by a trained research assistant who 
confirmed school eligibility, sought consent to participate 
in the study and scheduled a time for a telephone inter- 
view. At the completion of the telephone survey Principal 
consent was sought for observational data to be collected. 
The survey was conducted in late 2010. 

To identify a comprehensive list of school environment 
factors for inclusion in the telephone survey the research 
team conducted a review of the literature regarding school 
environmental factors associated with student diet, phys- 
ical activity, sedentary behaviour or excessive weight gain, 
as well as items from similar tools. A draft survey tool was 
circulated to an expert advisory group (n = 10) of health 
promotion practitioners, school education directors, local 
Principals and teachers, academics, parents and dietitians. 
The items were amended based on advisory group 
member's review and feedback regarding the scope of the 
survey, the appropriateness of survey items and compre- 
hension and then pilot tested with five primary school 
Principals to check acceptability and comprehension. The 
final survey tool (School Environment Assessment Tool 
(SEAT)) consisted of 65 survey items. A copy of the survey 
tool is available as supplementary material. 

Observational data 

Pre-service teachers on placement in primary schools in 
the region as part of their fourth year University teacher 
training program were recruited to collect the observa- 
tional data. Pre-service teachers were informed of the 
school practices to be observed at the beginning of their 
10-week placement. To ensure they had an adequate 
amount of time to observe all practices they were sur- 
veyed in week 9 of their 10 week placement, away from 
school premises to ensure that there was no influence 
from school personnel, using a paper-based form. The 
pre-service teachers were surveyed two weeks after the 
completion of the telephone survey by the school Princi- 
pal. They were not aware of the Principal's response to 
the telephone survey. If the pre service teachers did not 
observe any of the practices, for example fundraising ac- 
tivities, they were instructed to provide a 'don't know' 
response. The paper-based observational data collection 
form was developed using equivalent items from the 
telephone survey. The form was piloted with 15 pre- 
service teachers to establish its utility and acceptability, 
with minor amendments subsequently being made. 

Measures 
Food availability 

The SEAT required Principals to report (yes/ no/ don't 
know/ refused) if any of the following foods or drinks 



were sold at the school from the (a) canteen, (b) vending 
machines or (c) via fundraisers: fruit; vegetables; water; 
regular soft drinks (not diet); diet soft drinks; fruit juice; 
other sweetened drinks including cordials, energy drinks, 
flavoured mineral waters etc.; confectionary; sweet and 
savoury biscuits/ cakes/ muffins/ donuts or muesli bars; 
potato crisps or other salty snacks such as twisties or 
corn chips; deep fried foods; ice creams covered in choc- 
olate; other ice creams, iceblocks or frozen desserts. 

Physical activity opportunities 

The SEAT required Principals to report on the physical 
activity facilities and equipment and organised physical 
activities that students had available to them whilst at 
the school. These included; 

(a) Physical activity facilities and equipment- Principals 
were asked to report whether the school had any of 
the following that students can access everyday 
(yes/ no/ don't know) and if yes when can they access 
them (recess only/ lunch only/ both recess and lunch); 

- Large asphalt areas suitable for games e.g. 
basketball/ netball; 

- Large playing fields suitable for games e.g. soccer, 
football; 

- Indoor activity spaces e.g. multi-purpose rooms 
and gymnasiums; 

- Playground markings e.g. hopscotch, wall targets 
etc.; 

- Fixed playground equipment such as climbing 
structures, swings etc.; 

- Sports equipment provided by the school e.g. 
basketballs, netballs, skipping ropes etc. 

(b) Organised physical activities- Principals were asked 
if, during recess or lunch, the school had organised 
physical activities e.g. dance/ gymnastics (yes/ no/ 
don't know); whether they are co-ordinated by the 
school or an outside agency; and if teachers 
participate or join in. 

(c) Small screen recreation- Principals were asked to 
report whether students were able to access small 
screen recreation (yes/ no/ don't know) during 
recess or lunch; before or after school; during wet 
weather sport. 

(d) School sport- Principals were asked to report if 
both infants and primary students had sport. 

School characteristics 

During the telephone interview, Principals were asked to 
report the number of students attending the school. 
School type (Government or non-Government (Catholic 
or Independent) and the postcode of the locality of the 
school were obtained from school websites. 
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Analysis 

Statistical analyses were performed using SAS v9.2 (SAS 
Institute Inc., Cary, NC, USA). Descriptive statistics were 
used to describe school characteristics and the propor- 
tion of Principals and pre-service teachers who reported 
"yes" to any of the school practices. The reported num- 
ber of enrolled students in each school was used to cat- 
egorise school size as: 'small (1-159 students); 'medium' 
(160-450 students); or 'large' (451+ students) based on 
the Department of Education and Communities School 
Directory [30]. Schools with postcodes ranked socio- 
economically in the top 50% of New South Wales [31] 
were categorised as schools of 'higher socio-economic 
status' while those in the lower 50% were categorised as 
schools of 'lower socio-economic status'. School post- 
code areas were also used to categorise the school's 
locality as either 'rural' (those schools in outer regional, 
remote and very remote areas), or 'urban' (those in 
regional cities and inner regional areas) [32]. 

Two measures of agreement were calculated to 
describe the validity of Principal report. First, percent 
agreement between Principal report and observation was 
calculated for each item. As recommended, agreement 
levels >80% were considered 'strong' [33]. 

Whilst percent agreement is widely used and easily 
interpreted [34], it does not correct for agreement over and 
above what would be expected by chance [35]. As such, the 
second and primary measure of validity was assessed using 
the Kappa statistic [35]. Prevalence Adjusted and Bias 
Adjusted Kappa (PABAK) was calculated and reported 
where the prevalence of a particular school environment 
characteristics was found to be > 75% (or less than 25%). 
Non adjusted Kappa statistics were calculated and reported 
for items with prevalence between 26% and 74%. 
Benchmarks suggested by Landis and Koch [36] were used 
to classify agreement: < 0.00 = poor, 0.00 - 0.20 = slight, 
0.21 - 0.40 = fair, 0.41 - 0.60 = moderate, 0.61 - 0.80 = 
substantial and 0.81 - 1.0 = almost perfect. 

Results 

Sample 

In total, 384 of 403 Principals participated in the CATI. 
Of these, 63 schools had a pre-service teacher on place- 
ment in the Hunter New England Region of NSW and 
hence were eligible to participate. Of these, seven school 
Principals did not consent to having their pre-service 
teacher participate in data collection leaving a maximum 
potential sample of 56 (Principal consent rate = 89%). 
There was no statistically significant difference in terms 
of school size, socio-economic status or rurality for 
schools who consented to having their pre-service 
teachers complete the survey and those that did not. On 
the day of data collection with the pre-service teachers, 
three refused to participate and 11 were absent (pre- 



service teacher response rate = 75%). This provided 42 
matched pairs of CATI-observational data. 

Of the telephone survey respondents, 35 were Principals 
who had been in their position on average 5.9 years (SD 
4.0 years), with the remaining participants being Deputy or 
Assistant Principals (all respondents hereafter are referred 
to as Principals). The majority of the schools were medium 
sized (57%) (equal proportions of small and large schools 
participated (21%)), Government schools (67%), located in 
urban (98%) and in higher socio-economic areas (67%). 

Validity of Principal self report 

The validity of principal responses to the telephone survey 
items are presented in Table 1. As both Principals and 
observers reported that no vending machines were located 
in the schools, percent agreement and kappa scores were 
100% and are not presented for these 26 items. 

Percent agreement 

Per cent agreements for the remaining 39 items, ranged 
from 37% to 100%. Twenty items (51%) were found to 
have percent agreements greater than or equal to 80%. 
The questions related to student's access to small screen 
recreation had substantially lower per cent agreement 
estimates. 

Kappa statistics 

PABAK were calculated for 27 of the 39 items as they 
had prevalence > 75% (or less than 25%). Based on such 
analyses, three items had poor agreement (< 0.00), eight 
had slight agreement (0.00-0.20), one had fair agreement 
(0.21-0.40), seven had moderate agreement (0.41-0.60), 
11 had substantial agreement (0.61-0.80) and nine had 
almost perfect agreement (0.81-1.0). 

With respect to the food environment, foods sold 
through fundraisers had the highest Kappa/PABAK 
estimates with 10 items having substantial or better 
agreement. Kappa/ PABAK scores for foods sold in the 
canteen varied (range -0.6-0.81), although most had 
slight to moderate agreement. 

Across the four aspects of the physical activity envir- 
onment that were measured, Kappa/PABAK scores were, 
in general, quite good. Three items within "physical 
activity facilities" were found to have moderate or better 
agreement and two items within "organised physical 
activities" were found to have perfect agreement (school 
has supervised or instructed physical activity at recess 
and lunch and whether teacher's participate or join in 
the organised activities at recess and lunch). 

Discussion 

To the author's knowledge this is the first study to assess 
the validity of a Computer Assisted Telephone Interview 
(CATI) survey of principal reported primary school 
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Table 1 Validity of survey items assessing the nutrition and physical activity environment of primary schools 



Aspect of the school environment 



% Yes (Principal) 



1 Yes (Pre-service teacher) 



% Agree 



Kappa/PABAK (95% CI) 



Food sold in canteen 

• Fruit 

• Vegetables 
■ Water 

• Regular soft drinks 

• Diet soft drinks 

• Fruit juice 

• Other sweetened drinks 

• Confectionary 

• Sweet and savoury biscuits 

• Potato crisps 

• Deep fried foods 

• Ice creams covered in chocolate 

• Other ice creams 
Food sold in fundraising 

• Fruit 

• Vegetables 

• Water 

• Regular soft drinks 

• Diet soft drinks 

• Fruit juice 

• Other sweetened drinks 

• Confectionary 

• Sweet and savoury biscuits 

• Potato crisps 

• Deep fried foods 

• Ice creams covered in chocolate 

• Other ice creams 

Physical activity facilities accessible at recess and 

• Large asphalt areas 

• Large playing fields 

• Indoor activity spaces 

• Playground markings 

• Fixed playground equipment 

• School sports equipment 

Organised physical activities at recess and lunch 

• Organised physical activity 

• Do teachers join in 

Students access to small screen recreation 

• During Recess/Lunch 

• Before school 

• During wet weather sport 



95 
90 
93 
0 
2 



19 

55 
57 
0 
0 
95 

19 
24 
12 
17 
14 
17 
14 
31 
29 
12 
2 
0 
10 

lunch 

95 
83 
19 
95 
67 
60 

10 
40 

/ 

33 
5 



83 
62 
86 
24 
24 
93 
55 
43 
64 
69 
31 
21 



5 
7 
0 
/ 

2 
0 
/ 

12 
10 

/ 

7 

7 

5 

90 
88 
24 
90 
83 
93 

10 
40 

64 
19 
31 



79 
5/ 
88 
76 
74 
90 
69 
62 
52 
55 
69 
79 



81 

79 
88 
81 
88 
83 
88 
67 
67 
90 
95 
93 
90 

86 
81 
67 
90 
74 
57 

100 
100 

37 
59 
73 



0.57 (0.26, 0.79) a 
-0.06 (-0.26, 0.14) 
0.76 (0.49, 0.92) a 
0.52 (0.21, 0.76) a 
0.48 (0.16, 0.72) a 
0.81 (0.55, 0.95) a 
0.38 (0.11,0.66) 
0.16 (-0.1, 0.43) 
0.02 (-0.28, 0.32) 
0.04 (-0.25, 0.34) 

0 (0, 0) 
0.57 (0.26, 0.79) a 
0.76 (0.49, 0.92) a 

0.62 (0.32, 0.83) a 
0.57 (0.26, 0.79) a 
0.76 (0.49, 0.92) a 
0.62 (0.32, 0.83) a 
0.76 (0.49, 0.92) a 
0.67 (0.37, 0.86) a 
0.76 (0.49, 0.92) a 
0.06 (-0.21, 0.33) 
-0.02 (-0.26, 0.22) 
0.81 (0.55, 0.95) a 
0.90 (0.68, 0.99) a 
0.86 (0.61, 0.97) a 
0.81 (0.55, 0.95) a 

0.71 (0.43, 0.89) a 
0.62 (0.32, 0.83) a 
0.01 (-0.29, 0.32) 
0.81 (0.55, 0.95) a 
0.48 (0.16, 0.72) a 
-0.02 (-0.2, 0.15) 

1 (1, D a 
1 (1, D a 

0 (-0.12, 0.12) 
0.02 (-0.28, 0.31) 
0.46 (0.14, 0.72) a 
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Table 1 Validity of survey items assessing the nutrition and physical activity environment of primary schools 

(Continued) 

School sport 

• K-2 have sport 88 85 83 0.67 (0.37, 0.86) a 

• 3-6 have sport 95 90 90 0.81 (0.55, 0.95) a 
a Adjusted Kappa used if prevalence greater than or equal to 75% (or less than 25%); CI confidence interval. 



nutrition and physical activity environment characteristics 
using direct observation as the criterion standard. The 
findings of the study suggest that Principals can accurately 
report on a large number of food and physical activity 
environmental characteristics of their school. 

The study found that more than half of the survey items 
assessed had moderate, substantial or almost perfect 
agreement. Substantial or greater agreement was found 
for most items that assessed foods sold through 
fundraising, physical activity facilities of the school, and 
organised physical activities that occurred at recess and 
lunch. Such findings are consistent with the findings of 
the few other validation studies of the healthy eating and 
or physical activity and sports environment of schools 
[24,25]. The high level agreement found within these 
domains may be for a number of reasons. The physical 
activity facilities are permanent fixtures in the school and 
the organised activities occur regularly and are visible 
activities it is likely Principals can more accurately report 
on these. Similarly, as Principal approval is needed for 
school fundraising activities to occur, and the fact they 
may occur only once or twice a term, it is likely Principals 
are able to easily recall these events and the foods sold. 
However, these findings should be viewed with some 
caution, as no time period was specified, it is possible 
Principals may have been reporting on fundraising events 
throughout the year whilst pre-service teachers could only 
report on fundraising events which occurred during the 
nine weeks they were in the school. 

In contrast, foods sold through the canteen and students 
access to small screen recreation were found to have lower 
agreement. This is similar to findings of the Canadian 
validation study which found low agreement for school 
administrators and teachers being able to report on the 
schools provision of healthy vending machines [24]. As 
school canteens sell a wide variety of foods, and as products 
sold through canteens are frequently modified according to 
student demand and product availability and season, 
recalling foods sold through canteens may represent a 
significant challenge for Principals. Collection of school pol- 
icies and parent communication strategies (e.g. newsletter 
items) could be undertaken in future studies to help 
validate such items that are difficult to quantify by direct 
observation but are known to be associated with students 
physical activity and healthy eating such as school policies 
and parent engagement strategies [14]. 



A strength of this study was the use of observational 
data over a nine week period. However, this method- 
ology could be enhanced by tasking pre service teachers 
with observations prospectively rather than retrospect- 
ively. Given the differences in school systems, the 
generalizabilty of the SEAT to schools outside of 
Australia may be limited and thus may need to be 
adapted for international use and have its validity tested 
in these jurisdictions. Furthermore, given the challenges 
of collecting observational data the study utilised a 
convenience sample of pre-service teachers as observers. 
Nonetheless, randomisation of pre-service teachers to 
schools would have strengthened the design. 

Conclusion 

Given the significant influence schools have on children's 
behaviours this tool may provide researchers, policy 
makers and practitioners with an efficient method of 
assessing the nutrition and physical activity environment 
of primary schools and evidence as to whether obesity 
prevention interventions are achieving expected out- 
comes and the need for on-going intervention. 
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